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ABSTRACT 

Data-driven decision making, popularized in the 1980s and 1990s, is evolving into a vastly more 
sophisticated concept known as big data that relies on software approaches generally referred to as 
analytics. Big data and analytics for instructional applications are in their infancy and will take a few 
years to mature, although their presence is already being felt and should not be ignored. While big data 
and analytics are not panaceas for addressing all of the issues and decisions faced by higher education 
administrators, they can become part of the solutions integrated into administrative and instructional 
functions. The purpose of this article is to examine the evolving world of big data and analytics in 
American higher education. Specifically, it will look at the nature of these concepts, provide basic 
definitions, consider possible applications, and last but not least, identify concerns about their 
implementation and growth. 
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I. INTRODUCTION 

The title of Bob Dylan’s 1963 hit song, The Times They are A-Changin, is most appropriate as a 
description of the state of American higher education in the early part of the 21 st century. Besieged by 
mega-forces including a severe economic recession, globalization, increased government oversight, 
enrollment surges, decreases in public funding, calls for greater accountability, and tectonic shifts in the 
role of public, non-profit, and for profit institutions, American higher education is facing a significant 
crisis. Many of these forces are imposing serious stress on the financial stability and wherewithal of 
colleges and universities, mandating changes in their operation and administration. 

American higher education can respond to these forces and the changes they are setting in motion in 
several ways. It can wait and see and hope that the good old times will return. If they, in fact, ever 
existed in the recent past, it is not likely that the good times will be returning any time soon. It can 
consider fiscal exigency as a number of states and local governments have already done. This would 
result in severe programmatic cutbacks and reduce higher educational opportunities at a time when the 
demand is at an all-time high. Or, it can view this as a time of opportunity to examine and improve what it 
does. Indications are that many higher education institutions are moving in this third direction. This 
article advocates this third approach. 

Technology is at the center of much of the turbulence in our times. It will also be among the solutions 
that help us weather this period. The Internet has permeated every aspect of society and commerce by its 
ubiquity, and it has changed the world of higher education. Online and blended (partially online and 
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partially face-to-face) learning are changing the way instruction is provided in this country. More than 
six million students or approximately one-third of the higher education population enrolled in fully online 
college courses in 2010 [1], Millions more college students are enrolling in blended courses. Online 
learning has also spurred the growth of for-profit online colleges and universities, institutions that did not 
exist twenty years ago but now represent an important segment of the higher education community. The 
changes brought on by online access to instruction are also affecting how our colleges and universities are 
being administered. Infusions of technology infrastructure, large-scale databases, and demands for timely 
data to support decision making have seeped into all levels of college leadership and operations. Data- 
driven decision making, popularized in the 1980s and 1990s, is evolving into a vastly more sophisticated 
concept known as big data that relies on software approaches generally referred to as analytics. Big data 
and analytics for instructional applications are in their infancy and will take a few years to mature, 
although their presence is already being felt and should not be ignored. While big data and analytics are 
not panaceas for addressing all of the issues and decisions being faced by higher education administrators, 
they can become part of the solutions and integrated into administrative and instructional functions. The 
purpose of this article is to examine the evolving world of big data and analytics in American higher 
education. Specifically, it will look at the nature of these concepts, provide basic definitions, consider 
possible applications, and last but not least, identify concerns about their implementation and growth. 

II. BACKGROUND AND DEFINITIONS 

In many ways, American higher education has been at the forefront of digital technology since the 
introduction of computers in the 1950s. Much of the early research and development of digital computer 
systems occurred at major engineering schools such as the Massachusetts Institute of Technology, the 
University of Pennsylvania, Stanford University, and the University of Illinois at Urbana-Champaign. 
Some of this technology found its way into classrooms, laboratories, and eventually administration by the 
1960s, when most American colleges started to use technology to maintain administrative records on 
finances, students, and personnel. These early applications were rudimentary by today’s standards, using 
Hollerith (punched) cards, sequential magnetic tape files, and large mainframe computers to collect and 
store data. In the late 1960s and 1970s, many administrative applications migrated to direct access 
magnetic disk-storage technology that fueled the development of online recordkeeping applications. In 
the late 1970s and 1980s, minicomputers and microcomputers further changed the way many 
administrative applications operated, as a good deal of processing was moved off mainframes to the 
smaller hardware. In the 1990s, the Internet again changed the technology world for everybody as 
administrative applications moved to web-based interfaces and more sophisticated software technology. 
In the early part of the 21 st century, social networking and mobile technology moved the Internet into a 
twenty-four hour, on-demand companion for much of what we do. 

Over time, administrative decision making evolved as well, as more data were made available from 
integrated information systems that could dabble in “what if’ questions using database query languages 
and decision-support systems. The responsibilities of institutional research offices changed from 
conducting static yearly studies to culling information from the institution’s database management 
systems on an on-going basis. Regional accrediting bodies began to require colleges to demonstrate a 
command of the information in their institutions and demanded evidence of data-informed rational 
planning and decision processes. Most colleges have been able to meet these requirements and have 
integrated technology into these processes. 

In the 1990s and the early 2000s, a new phenomenon generally termed online learning emerged that has 
changed the way faculty teach and students learn. As mentioned earlier, millions of students are learning 
online, and entire colleges have been “built” that offer the entirety of their academic programs online. 
This phenomenon has opened up new approaches and avenues for collecting and processing data on 
students and course activities; every instructional transaction can be immediately recorded and added to a 
database. Academic administration, which in the past occurred away from the classroom, can now be 
integrated very closely with instructional activities and requires close collaboration with the teaching 
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faculty. 

Technology is prone to developing terminology that is uniquely suited to specific situations, and 
technology used in administration and management is no exception. The focus of this article is 
technology-based approaches that support decision making in higher education. The simplest definition of 
the popular term “data-driven decision making” is the use of data analysis to inform courses of action 
involving policy and procedures. Inherent in this definition is the development of reliable and timely 
information resources to collect, sort, and analyze the data used in the decision making process. It is 
important to note that data analysis is used to inform and does not mean to replace entirely the experience, 
expertise, intuition, judgment, and acumen of competent educators. While decision making may be singly 
defined as choosing between or among two or more alternatives, in a modem educational organization, 
decision making is an integral component of complex management processes such as academic planning, 
policy making, and budgeting. These processes evolve over time, require participation by stakeholders, 
and most importantly, seek to include information which will help all those involved in the decision 
process. 

Fundamental to data-driven decision making is a rational model directed by values and based on data. It 
is well-recognized, however, that a strictly rational model has limitations. An individual commonly 
associated with this concept and whose work is highly recommended for further reference is Herbert 
Simon [2-6]. Simon was awarded the Nobel Prize in economics in 1978 for his research on decision 
making in organizations. His theory on the limits of rationality, later renamed “bounded rationality,” has 
as its main principle that organizations operate along a continuum of rational and social behaviors mainly 
because the knowledge necessary to function strictly according to a rational model is beyond what is 
available. Although first developed in the 1940s, this theory has withstood the test of time and is widely 
recognized as a fundamental assumption in understanding organizational processes such as decision 
making and planning [7-9]. On the other hand, modem computerized information systems are facilitating 
and instilling a greater degree of rationality in decision making in all organizations including colleges and 
universities. They support organizations and help them to adjust, adapt, and leam in order to perform 
their administrative functions [10]. While these systems are not replacing the decision maker, they surely 
are helping to refine the decision-making process. 

Figure 1 (below) illustrates the basic data-driven decision-making process. It assumes that an information 
system is available to support the decision process, that internal and external factors not available through 
the information system are considered, and that a course or courses of action are determined. The 
information system in Figure 1 is a computerized database system capable of storing, manipulating, and 
providing reports from a wide variety of data. 

Terms related to data-driven decision making include data warehousing, data mining, and data 
disaggregation. Data warehousing essentially refers to a database information system that is capable of 
storing, integrating and maintaining large amounts of data over time. It might also involve multiple 
database systems. Data mining is a frequently used term in research and statistics which refers to 
searching or “digging into” a data file for information to understand better a particular phenomenon. Data 
disaggregation refers to the use of software tools to break data files down into various characteristics. An 
example might be using a software program to select student performance data by gender, by major, by 
ethnicity, or by other definable characteristics. 
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Figure 1. The Data-Driven Decision-Making Process 

In recent years, two other terms, big data and analytics, have grown in popularity. Big data is a generic 
term that assumes that the information or database system(s) used as the main storage facility is capable 
of storing large quantities of data longitudinally and down to very specific transactions. For example, 
college student record keeping systems have maintained outcomes information on students such as grades 
in each course. This information could be used by institutional researchers to study patterns of student 
performance over time, usually from one semester to another or from one year to another. In a big data 
scenario, data would be collected for each student transaction in a course, especially if the course was 
delivered electronically online. Every student entry on a course assessment, discussion board entry, blog 
entry, or wiki activity could be recorded, generating thousands of transactions per student per course. 
Furthermore, this data would be collected in real or near real time as it is transacted and then analyzed to 
suggest courses of action. Analytics software is evolving to assist in this analysis. 

The generic definition of analytics is similar to data-driven decision making. Essentially it is the science 
of examining data to draw conclusions and, when used in decision making, to present paths or courses of 
action. In recent years, the definition of analytics has gone further, however, to incorporate elements of 
operations research such as decision trees and strategy maps to establish predictive models and to 
determine probabilities for certain courses of action. It uses data mining software to establish decision 
processes that convert data into actionable insight, uncover patterns, alert and respond to issues and 
concerns, and plan for the future. This might seem to be an overly complicated definition, but the term 
“analytics” has been used in many different ways in recent years and has become part of the buzzword 
jargon that sometimes seeps into new technology applications and products. Goldstein and Katz in a 
study of academic analytics admitted that they struggled with coming up with a name and definition that 
was appropriate for their work. They stated that they adopted the term “academic analytics” for their 
study but that it was an “imperfect label” [11]. Alias defined four different types of analytics that could 
apply to instruction including web analytics, learning analytics, academic analytics and action analytics 
[12]. The trade journal, Infoworld, referred to analytics as: 

One of the buzzwords around business intelligence software...[that]...has been through the 

linguistic grinder, with vendors and customers using it to describe very different functions. 

The term can cause confusion for enterprises, especially as they consider products from vendors 
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who use analytics to mean different things.. .[13] 

What is critical in defining analytics is the use of data to determine courses of action especially where 
there is a high volume of transactions. Common examples of analytics applications are when ecommerce 
companies such as amazon.com or Netflix examine Web site traffic, purchases, or navigation patterns to 
determine which customers are more or less likely to buy particular products (i.e., book, movie). Using 
these patterns, companies send notifications to customers of new products as they become available. In 
higher education, analytics are beginning to be used for a number of applications that address student 
performance, outcomes, and persistence. 

III. APPLICATIONS 

Big data concepts and analytics can be applied to a variety of higher education administrative and 
instructional applications, including recruitment and admissions processing, financial planning, donor 
tracking, and student performance monitoring. In keeping with the theme of this special edition of JALN, 
the applications discussed in this article will focus on teaching and learning and, hence, will specifically 
examine learning analytics. 

To take advantage of big data and learning analytics, it is almost a requirement that transaction processing 
be electronic rather than manual. Traditional face-to-face instruction can support traditional data-driven 
decision-making processes, however, to move into the more extensive and especially time-sensitive 
learning analytics applications, it is important that instructional transactions are collected as they occur. 
This would be possible in the case of a course management/leaming management system (CMS/LMS). 
Most CMSs provide constant monitoring of student activity whether they are responses, postings on a 
discussion board, accesses to reading material, completions of a quiz, or some other assessment. Using 
the full capabilities of a basic CMS, a robust fifteen-week online course could generate thousands of 
transactions per student. Real-time recording and analysis of these transactions can be used to feed a 
learning analytics application. Critical to this type of application is not waiting learning analytics software 
application. The instructional transactions should also be integrated with other resources such as data 
from the college information systems (student, course, faculty) and an analytics software program. The 
logic/decision trees for the latter are based on patterns as well as faculty and adviser experiences, intuition 
and insights that are used to develop guidelines and rules for subsequent courses of action (see Figure 2). 
One important caveat is that the data accuracy should never be compromised in favor of timeliness of the 
data, both for accuracy and for the end of a marking period or semester to record performance measures. 
The reason this is important is that monitoring student transactions on a real-time basis allows for real¬ 
time alerts. Instructors may take actions or intervene in time to assist students. A CMS or something 
similar therefore becomes critical for collecting and feeding this data into a “big” database for processing 
by timeliness are important and need to be present in the learning analytics application. 
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Figure 2. Learning Analytics Flow Model 

In a white paper published by IBM entitled Analytics for Achievement, eight categories of instructional 
applications were described. While developed for education in general, they are nevertheless appropriate 
for the discussion here. The eight categories are as follows: 

1. Monitoring individual student performance 

2. Disaggregating student performance by selected characteristics such as major, year of 
study, ethnicity, etc. 

3. Identifying outliers for early intervention 

4. Predicting potential so that all students achieve optimally 

5. Preventing attrition from a course or program 

6. Identifying and developing effective instructional techniques 

7. Analyzing standard assessment techniques and instruments (i.e. departmental and 
licensing exams) 

8. Testing and evaluation of curricula. [14] 

Of the above, monitoring individual student performance and participation in a course is among the most 
popular type of learning analytics application. Anyone who has ever taught (face-to-face or online) will 
frequently monitor student participation to determine engagement with the course material. Taking 
attendance is a time-honored activity, and most instructors will become concerned about students who 
have too many absences. Grades on quizzes and papers are also frequently monitored. A conscientious 
instructor will review his/her records and meet with those students who are not meeting some standards 
for the course. Many colleges have instituted mid-term reviews that provide students with indicators of 
their progress in a course. In online courses, CMSs routinely provide course monitoring statistics and 
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rudimentary early warning systems that allow instructors to follow up with students who are not 
responding on blogs or discussion boards, not accessing reading materials, or not promptly taking 
quizzes. These course statistics are maintained in real-time, and instructors can review them as often they 
wish. Again, students who are not as engaged as they should be can be sent an email expressing concerns 
about their performance. None of these interventions require learning analytics; however, this approach 
can be enhanced significantly by expanding the amount and nature of the data collected. For example, a 
single student response on a discussion board can be analyzed for patterns to determine the depth and 
quality of student engagement with the course material. These patterns are uncovered by examining 
thousands and tens of thousands of other student responses and evaluating sentences and phrases. 

Student attrition/retention (see Table 1 for graduation rates at American colleges and universities) has 
been a significant issue in higher education for decades [15]. Graduation rates at the public four-year 
institutions are 55.7%; 65.1% at the four-year private non-profits; and 20.4% the four-year for-profits. 
Rates at the two-year institutions are 22.1% at the public institutions; 55.3% at the private non-profits; 
and 60.9% at the private for-profits. It needs to be mentioned that graduation rates at the two-year 
institutions include certificate programs, many of which are of shorter duration (less than two years) to 
completion. Some in fact are only a few months in duration. Private two-year institutions, especially the 
for-profits, enroll much larger percentages of students into certificate programs than do public 
institutions. Student attrition/retention is receiving significant attention at the U.S. Department of 
Education and is increasingly becoming the focus of a major initiative in President Barack Obama’s 
administration. Several for-profit colleges have come under the microscope in recent years for abuses of 
financial aid and extremely poor retention rates. The issue, while not necessarily the result of abuse, is 
not unique to for-profit institutions and all of higher education needs to pay attention to retention and 
attrition. Student attrition is not a simple phenomenon and involves a host of variables related to the 
academic and social integration of students into a college program. The work of Vincent Tinto is highly 
recommended for readers wishing more background information on student attrition models [16]. 
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Table 1. Higher Education Graduation Rates - 2003 Cohort 

Source: U.S. Department of Education, National Center for Education Statistics. The Condition of Education 2011 (NCES 

2011-033). 


An appropriate learning analytics application was developed at Rio Salado Community College in 
Arizona. Rio Salado enrolls more than 41,000 students in online courses. One of its “instructional 
priorities includes a strong emphasis on personalization—helping nontraditional students reach their 
educational goals through programs and services tailored to individual needs” [17]. To achieve this 
personalization, the college has implemented advisement and instructional systems including the Progress 
and Course Engagement (PACE) system for automated tracking of student progress—with intervention as 
needed. PACE is an analytics application. To develop PACE, Michael Cottam, associate dean for 
instructional design at Rio Salado, indicated that: 

[We] crunched data from tens of thousands of students, we found that there are three main 
predictors of success: the frequency of a student logging into a course; site engagement—whether 
they read or engage with the course materials online and do practice exercises and so forth; and 
how many points they are getting on their assignments. All that may sound simple, but the 
statistics we encounter are anything but simple. And we've found that, overwhelmingly, these 
three factors do act as predictors of success... 

The reports we generate show green, yellow, and red flags—like a traffic light—so that 
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instructors can easily see who is at risk. We can predict, after the first week of a course, with 70 
percent accuracy, whether any given student will complete the course successfully (with a grade 
of “C” or better). That's our “eighth day” at risk model. A second model includes weekly updates 
using similar predictive factors. [17] 

An instructor can review student engagement at any time throughout the course and data in PACE is 
maintained on a real-time basis. At the time of this writing, PACE was being pilot-tested and was 
scheduled for full deployment throughout the college in 2012. Student performance, especially as related 
to attrition and retention, has emerged as a primary focus of learning analytics applications. Several other 
colleges such as Purdue University and American Public University System have or are developing 
similar applications. 

There are a number of other existing learning analytics applications in use today. Examples include the 
following: 

•Northern Arizona University has an early warning alert and retention system called Grade 
Performance System (GPS). Students receive alerts regarding grades, attendance, academic issues as 
well as positive feedback. Students have a number of options and resources to use depending upon 
the nature of the alert. For more information: http://www4.nau.edu/ua/GPS/student/. 

•Purdue University has a Course Signals System designed to increase student success in the 
classroom. Purdue University's Course Signals application detects early warning signs and provides 
intervention to students who may not be performing to the best of their abilities before they reach a 
critical point. Course Signals has three important characteristics: 

•It provides real-time feedback 

•Interventions start early - as early as the second week of class 
•It provides frequent and ongoing feedback 
For more information: http://www.itap.purdue.edu/leaming/tools/signals/ 

•Ball State University has developed a Visualizing Collaborative Knowledge Work learning 
analytics application to enhance collaborative knowledge-building activities, by leveraging modem 
theories of rhetoric and writing, human-computer interaction, and the science of learning. 
Specifically, it uses techniques of interaction design and information visualization to encourage 
continuous formative evaluation and to promote metacognition among collaborators. For more 
information: http://emergingmediainitiative.com/project/leaming-analytics/ 

The above systems are fairly sophisticated and took a long time to develop, test, and implement. 
Educators who are considering developing a learning analytics application would be wise to consult with 
others who have already started using this technology. 

IV. CONCERNS 

The New Horizon Report is published each year by The New Media Consortium and EDUCAUSE. It 
predicts six emerging technologies that are likely “to enter mainstream use” over the next five years. In 
the 2012 Report, the six technologies in ra nk order were identified as follows: 

1. Mobile Applications 

2. Tablet Computing 

3. Game-Based Learning 

4. Learning Analytics 

5. Gesture-Based Learning 

6. Internet of Things [18] 

The ranking of these six technologies indicates that the first two will likely enter the mainstream in one 
year; the second two within two or three years; and the last two within five years. Learning analytics was 
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ra nk ed fourth and indicates that regular use and adoption of this technology is projected to be about two 
or three years away. This ranking is appropriate mainly because implementing and deriving significant 
benefit from learning analytics is still being explored at this time. There are also a number of concerns 
about this emerging technology that need to be considered. 

First, in order for big data and learning analytics applications to function well, data need to be accurate 
and timely. As indicated earlier in this article, learning analytics applications will work best for courses 
that are delivered electronically. While almost one-third of all college students take at least one online 
course every year, about two-thirds do not. Traditional face-to-face courses that require significant data 
conversion time will likely render this approach problematic. As a result, investing and developing 
highly-sophisticated big data and analytics applications for colleges that deliver the vast majority of their 
instruction in traditional face-to-face mode should be carefully considered to determine the costs and 
benefits. Courses that are not teacher-led but mediated substantially by software such as those developed 
as part of Carnegie Mellon University’s Open Learning Initiative (OL1) would be more conducive to an 
analytics application because instructional transactions are entirely electronically monitored and 
controlled by software [19]. 

Second, there are not yet enough individuals trained to use big data and analytics appropriately. 
Experienced database administrators and designers capable of warehousing and integrating data across 
multiple files and formats are a necessity. In addition to the expertise needed to develop databases, 
instructional designers working with faculty will need to understand and derive insights into the student 
behaviors that are pertinent to the application at hand. There is also a need for institutional researchers or 
others knowledgeable about statistics, decision trees, and strategy mapping to develop algorithms to 
construct predictive models. College administrators may have to invest in consultants or undertake 
extensive professional development with their own staffs in order to develop appropriate applications. 
This will take time and additional resources and may or may not be worth the return on investment. 
Furthermore, because of a dearth of expertise, there may also be a tendency to use instructional templates 
that are integrated into CMSs. These, although convenient, may be overly simplistic and considered with 
caution. 

Third, and perhaps the most serious concern, is that since learning analytics requires massive amounts of 
data collected on students and integrated with other databases, colleges need to be careful about privacy, 
data profiling, and the rights of students in terms of recording their individual behaviors. While college 
classes have always involved evaluating student performance and academic behavior, learning analytics 
takes the recording of behavior to a whole other level and scope, and needs to be evaluated. As well- 
intentioned as learning analytics might be in terms of helping students succeed, this “big data” approach 
may also be seen as “big brother is watching” and as an invasion of privacy that some students would 
rather not have imposed upon them. Precautions need to be taken to ensure that the extensive data 
collections of student instructional transactions are not abused in ways that potentially hurt individuals. 
Fourth, there is also a concern that a good deal of college and university student data may end up in larger 
governmental databases either at the state or national level. Bennett cautions that the United States is 
heading to an all-inclusive national K-20 database [20]. Federal education policies as promulgated by No 
Child Left Behind and Race to the Top funding have pushed many states to adopt comprehensive 
statewide student databases that could easily be the basis for establishing a national system. Furthermore, 
there is a certain amount of influence being exerted on the part of the U.S. Department of Education in 
favor of development of common database structures. Such a system might be beneficial but may also 
leave individuals vulnerable to privacy, data security, and theft issues. In 2011, the well-publicized 
scandal at The News Corporation in the United Kingdom where messages of a deceased child were 
hacked and modified by unscrupulous reporters and operatives cannot be forgotten. This was not the act 
of a rogue or radical individual but the common practice of the multi-billion dollar international News 
Corporation, owned by Rupert Murdoch (Fox News, New York Post). The users of “big data” and 
analytics need to be careful that these mega-database systems do not become a playground for deceitful 
and exploitative individuals and organizations. 
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Lastly, before concluding this section, it might be beneficial to return to Herbert Simon’s theory on the 
limits of rational decision making discussed earlier in this article. As database systems become “bigger” 
and the software such as analytics becomes more sophisticated, a case can be made that the limits of 
rational decision making are being stretched further because of the plethora of information and data 
available. Simon was also highly focused on the efficient use of data and is famously quoted as saying: 
“...a wealth of information creates a poverty of attention...” [21] Simon’s quote is a most appropriate 
concern in the era of big data and analytics. 

V. CONCLUSION 

This article started with a reference to Bob Dylan’s hit The Times They are A-Changin, and it is 
appropriate to end with it also. Colleges and universities need to meet a number of challenges that have 
already started to impact the elite position that American higher education has enjoyed for decades [22]. 
Some of these challenges are global and external to the higher education enterprise. As an example, a 
deep, extended recession has caused stress on the entire society especially on employment and state 
finances. Higher education has to rely on the political and economic sectors of the country to resolve this 
complex situation. However, there are issues that higher education can address in terms of expanding 
educational opportunity and thereby improving employment opportunity that go beyond opening doors 
for students. Higher education must strive to ensure that access means students can complete degrees. 
Online learning is part of the solution to this problem but student attrition in colleges and universities is at 
unacceptable rates and needs to be addressed as well. Data-driven decision making is already being used 
to help colleges identify and evaluate strategies that can improve retention. As data-driven decision 
making enters the big data and learning analytics era, these new approaches, while not silver bullets, may 
be part of the solution. Higher education administrators will do well by evaluating whether they can be 
used in their institutions and determining the role they can play. 
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